

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>设备管理 &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/css/custom.css" type="text/css" />

  
  
    <link rel="shortcut icon" href="../../../_static/favicon.ico"/>
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/jquery.js"></script>
        <script src="../../../_static/underscore.js"></script>
        <script src="../../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../genindex/" />
    <link rel="search" title="Search" href="../../../search/" />
    <link rel="next" title="迁移到 BlueStore" href="../bluestore-migration/" />
    <link rel="prev" title="增加/删除监视器" href="../add-or-rm-mons/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../../" class="icon icon-home"></a> &raquo;</li>
        
          <li><a href="../../">Ceph 存储集群</a> &raquo;</li>
        
          <li><a href="../">集群运维</a> &raquo;</li>
        
      <li>设备管理</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="../../../_sources/rados/operations/devices.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../">
          

          
            
            <img src="../../../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../start/intro/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephadm/">Cephadm</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../">Ceph 存储集群</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../configuration/">配置</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">运维</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../operating/">操纵集群</a></li>
<li class="toctree-l3"><a class="reference internal" href="../health-checks/">健康检查</a></li>
<li class="toctree-l3"><a class="reference internal" href="../monitoring/">监控集群</a></li>
<li class="toctree-l3"><a class="reference internal" href="../monitoring-osd-pg/">监控 OSD 和归置组</a></li>
<li class="toctree-l3"><a class="reference internal" href="../user-management/">用户管理</a></li>
<li class="toctree-l3"><a class="reference internal" href="../pg-repair/">修复 PG 不一致状态</a></li>
<li class="toctree-l3"><a class="reference internal" href="../data-placement/">数据归置概览</a></li>
<li class="toctree-l3"><a class="reference internal" href="../pools/">存储池</a></li>
<li class="toctree-l3"><a class="reference internal" href="../erasure-code/">纠删码</a></li>
<li class="toctree-l3"><a class="reference internal" href="../cache-tiering/">分级缓存</a></li>
<li class="toctree-l3"><a class="reference internal" href="../placement-groups/">归置组</a></li>
<li class="toctree-l3"><a class="reference internal" href="../balancer/">均衡器</a></li>
<li class="toctree-l3"><a class="reference internal" href="../upmap/">使用 pg-upmap</a></li>
<li class="toctree-l3"><a class="reference internal" href="../crush-map/">CRUSH 图</a></li>
<li class="toctree-l3"><a class="reference internal" href="../crush-map-edits/">手动编辑一个 CRUSH 图</a></li>
<li class="toctree-l3"><a class="reference internal" href="../stretch-mode/">Stretch Clusters</a></li>
<li class="toctree-l3"><a class="reference internal" href="../change-mon-elections/">Configure Monitor Election Strategies</a></li>
<li class="toctree-l3"><a class="reference internal" href="../add-or-rm-osds/">增加/删除 OSD</a></li>
<li class="toctree-l3"><a class="reference internal" href="../add-or-rm-mons/">增加/删除监视器</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">设备管理</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#id2">设备跟踪</a></li>
<li class="toctree-l4"><a class="reference internal" href="#identifying-physical-devices">Identifying physical devices</a></li>
<li class="toctree-l4"><a class="reference internal" href="#enabling-monitoring">打开监控</a></li>
<li class="toctree-l4"><a class="reference internal" href="#scraping">Scraping</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id4">失败预测</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id5">健康警报</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id6">自动化的迁移</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../bluestore-migration/">迁移到 BlueStore</a></li>
<li class="toctree-l3"><a class="reference internal" href="../control/">命令参考</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/community/">Ceph 社区</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/troubleshooting-mon/">监视器故障排除</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/troubleshooting-osd/">OSD 故障排除</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/troubleshooting-pg/">归置组排障</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/log-and-debug/">日志记录和调试</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/cpu-profiling/">CPU 剖析</a></li>
<li class="toctree-l3"><a class="reference internal" href="../../troubleshooting/memory-profiling/">内存剖析</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../man/">    手册页</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../troubleshooting/">故障排除</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../api/">APIs</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <div class="section" id="devices">
<span id="id1"></span><h1>设备管理<a class="headerlink" href="#devices" title="Permalink to this headline">¶</a></h1>
<p>Ceph tracks which hardware storage devices (e.g., HDDs, SSDs) are consumed by
which daemons, and collects health metrics about those devices in order to
provide tools to predict and/or automatically respond to hardware failure.</p>
<div class="section" id="id2">
<h2>设备跟踪<a class="headerlink" href="#id2" title="Permalink to this headline">¶</a></h2>
<p>You can query which storage devices are in use with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">ls</span>
</pre></div>
</div>
<p>You can also list devices by daemon or by host:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">ls</span><span class="o">-</span><span class="n">by</span><span class="o">-</span><span class="n">daemon</span> <span class="o">&lt;</span><span class="n">daemon</span><span class="o">&gt;</span>
<span class="n">ceph</span> <span class="n">device</span> <span class="n">ls</span><span class="o">-</span><span class="n">by</span><span class="o">-</span><span class="n">host</span> <span class="o">&lt;</span><span class="n">host</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>For any individual device, you can query information about its
location and how it is being consumed with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">info</span> <span class="o">&lt;</span><span class="n">devid</span><span class="o">&gt;</span>
</pre></div>
</div>
</div>
<div class="section" id="identifying-physical-devices">
<h2>Identifying physical devices<a class="headerlink" href="#identifying-physical-devices" title="Permalink to this headline">¶</a></h2>
<p>You can blink the drive LEDs on hardware enclosures to make the replacement of
failed disks easy and less error-prone.  Use the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">device</span> <span class="n">light</span> <span class="n">on</span><span class="o">|</span><span class="n">off</span> <span class="o">&lt;</span><span class="n">devid</span><span class="o">&gt;</span> <span class="p">[</span><span class="n">ident</span><span class="o">|</span><span class="n">fault</span><span class="p">]</span> <span class="p">[</span><span class="o">--</span><span class="n">force</span><span class="p">]</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">&lt;devid&gt;</span></code> parameter is the device identification. You can obtain this
information using the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">ls</span>
</pre></div>
</div>
<p>The <code class="docutils literal notranslate"><span class="pre">[ident|fault]</span></code> parameter is used to set the kind of light to blink.
By default, the <cite>identification</cite> light is used.</p>
<div class="admonition note">
<p class="admonition-title">Note</p>
<p>This command needs the Cephadm or the Rook <a class="reference external" href="https://docs.ceph.com/docs/master/mgr/orchestrator/#orchestrator-cli-module">orchestrator</a> module enabled.
The orchestrator module enabled is shown by executing the following command:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">orch</span> <span class="n">status</span>
</pre></div>
</div>
</div>
<p>The command behind the scene to blink the drive LEDs is <cite>lsmcli</cite>. If you need
to customize this command you can configure this via a Jinja2 template:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">config</span><span class="o">-</span><span class="n">key</span> <span class="nb">set</span> <span class="n">mgr</span><span class="o">/</span><span class="n">cephadm</span><span class="o">/</span><span class="n">blink_device_light_cmd</span> <span class="s2">&quot;&lt;template&gt;&quot;</span>
<span class="n">ceph</span> <span class="n">config</span><span class="o">-</span><span class="n">key</span> <span class="nb">set</span> <span class="n">mgr</span><span class="o">/</span><span class="n">cephadm</span><span class="o">/&lt;</span><span class="n">host</span><span class="o">&gt;/</span><span class="n">blink_device_light_cmd</span> <span class="s2">&quot;lsmcli local-disk-{{ ident_fault }}-led-{{&#39;on&#39; if on else &#39;off&#39;}} --path &#39;{{ path or dev }}&#39;&quot;</span>
</pre></div>
</div>
<p>The Jinja2 template is rendered using the following arguments:</p>
<ul class="simple">
<li><dl class="simple">
<dt><code class="docutils literal notranslate"><span class="pre">on</span></code></dt><dd><p>A boolean value.</p>
</dd>
</dl>
</li>
<li><dl class="simple">
<dt><code class="docutils literal notranslate"><span class="pre">ident_fault</span></code></dt><dd><p>A string containing <cite>ident</cite> or <cite>fault</cite>.</p>
</dd>
</dl>
</li>
<li><dl class="simple">
<dt><code class="docutils literal notranslate"><span class="pre">dev</span></code></dt><dd><p>A string containing the device ID, e.g. <cite>SanDisk_X400_M.2_2280_512GB_162924424784</cite>.</p>
</dd>
</dl>
</li>
<li><dl class="simple">
<dt><code class="docutils literal notranslate"><span class="pre">path</span></code></dt><dd><p>A string containing the device path, e.g. <cite>/dev/sda</cite>.</p>
</dd>
</dl>
</li>
</ul>
</div>
<div class="section" id="enabling-monitoring">
<span id="id3"></span><h2>打开监控<a class="headerlink" href="#enabling-monitoring" title="Permalink to this headline">¶</a></h2>
<p>Ceph can also monitor health metrics associated with your device.  For
example, SATA hard disks implement a standard called SMART that
provides a wide range of internal metrics about the device’s usage and
health, like the number of hours powered on, number of power cycles,
or unrecoverable read errors.  Other device types like SAS and NVMe
implement a similar set of metrics (via slightly different standards).
All of these can be collected by Ceph via the <code class="docutils literal notranslate"><span class="pre">smartctl</span></code> tool.</p>
<p>You can enable or disable health monitoring with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">monitoring</span> <span class="n">on</span>
</pre></div>
</div>
<p>or:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">monitoring</span> <span class="n">off</span>
</pre></div>
</div>
</div>
<div class="section" id="scraping">
<h2>Scraping<a class="headerlink" href="#scraping" title="Permalink to this headline">¶</a></h2>
<p>如果打开了监控，指标会定期自动 scrape 。这个间隔时间可以这样配置：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">config</span> <span class="nb">set</span> <span class="n">mgr</span> <span class="n">mgr</span><span class="o">/</span><span class="n">devicehealth</span><span class="o">/</span><span class="n">scrape_frequency</span> <span class="o">&lt;</span><span class="n">seconds</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>The default is to scrape once every 24 hours.</p>
<p>You can manually trigger a scrape of all devices with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">scrape</span><span class="o">-</span><span class="n">health</span><span class="o">-</span><span class="n">metrics</span>
</pre></div>
</div>
<p>A single device can be scraped with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">scrape</span><span class="o">-</span><span class="n">health</span><span class="o">-</span><span class="n">metrics</span> <span class="o">&lt;</span><span class="n">device</span><span class="o">-</span><span class="nb">id</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>Or a single daemon’s devices can be scraped with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">scrape</span><span class="o">-</span><span class="n">daemon</span><span class="o">-</span><span class="n">health</span><span class="o">-</span><span class="n">metrics</span> <span class="o">&lt;</span><span class="n">who</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>The stored health metrics for a device can be retrieved (optionally
for a specific timestamp) with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">get</span><span class="o">-</span><span class="n">health</span><span class="o">-</span><span class="n">metrics</span> <span class="o">&lt;</span><span class="n">devid</span><span class="o">&gt;</span> <span class="p">[</span><span class="n">sample</span><span class="o">-</span><span class="n">timestamp</span><span class="p">]</span>
</pre></div>
</div>
</div>
<div class="section" id="id4">
<h2>失败预测<a class="headerlink" href="#id4" title="Permalink to this headline">¶</a></h2>
<p>Ceph can predict life expectancy and device failures based on the
health metrics it collects.  There are three modes:</p>
<ul class="simple">
<li><p><em>none</em>: disable device failure prediction.</p></li>
<li><p><em>local</em>: use a pre-trained prediction model from the ceph-mgr daemon</p></li>
</ul>
<p>The prediction mode can be configured with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">config</span> <span class="nb">set</span> <span class="k">global</span> <span class="n">device_failure_prediction_mode</span> <span class="o">&lt;</span><span class="n">mode</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>Prediction normally runs in the background on a periodic basis, so it
may take some time before life expectancy values are populated.  You
can see the life expectancy of all devices in output from:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">ls</span>
</pre></div>
</div>
<p>You can also query the metadata for a specific device with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">info</span> <span class="o">&lt;</span><span class="n">devid</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>You can explicitly force prediction of a device’s life expectancy with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">predict</span><span class="o">-</span><span class="n">life</span><span class="o">-</span><span class="n">expectancy</span> <span class="o">&lt;</span><span class="n">devid</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>If you are not using Ceph’s internal device failure prediction but
have some external source of information about device failures, you
can inform Ceph of a device’s life expectancy with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="nb">set</span><span class="o">-</span><span class="n">life</span><span class="o">-</span><span class="n">expectancy</span> <span class="o">&lt;</span><span class="n">devid</span><span class="o">&gt;</span> <span class="o">&lt;</span><span class="n">from</span><span class="o">&gt;</span> <span class="p">[</span><span class="o">&lt;</span><span class="n">to</span><span class="o">&gt;</span><span class="p">]</span>
</pre></div>
</div>
<p>Life expectancies are expressed as a time interval so that
uncertainty can be expressed in the form of a wide interval. The
interval end can also be left unspecified.</p>
</div>
<div class="section" id="id5">
<h2>健康警报<a class="headerlink" href="#id5" title="Permalink to this headline">¶</a></h2>
<p>The <code class="docutils literal notranslate"><span class="pre">mgr/devicehealth/warn_threshold</span></code> controls how soon an expected
device failure must be before we generate a health warning.</p>
<p>The stored life expectancy of all devices can be checked, and any
appropriate health alerts generated, with:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">device</span> <span class="n">check</span><span class="o">-</span><span class="n">health</span>
</pre></div>
</div>
</div>
<div class="section" id="id6">
<h2>自动化的迁移<a class="headerlink" href="#id6" title="Permalink to this headline">¶</a></h2>
<p>If the <code class="docutils literal notranslate"><span class="pre">mgr/devicehealth/self_heal</span></code> option is enabled (it is by
default), then for devices that are expected to fail soon the module
will automatically migrate data away from them by marking the devices
“out”.</p>
<p>The <code class="docutils literal notranslate"><span class="pre">mgr/devicehealth/mark_out_threshold</span></code> controls how soon an
expected device failure must be before we automatically mark an osd
“out”.</p>
</div>
</div>



           </div>
           
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="../bluestore-migration/" class="btn btn-neutral float-right" title="迁移到 BlueStore" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
        <a href="../add-or-rm-mons/" class="btn btn-neutral float-left" title="增加/删除监视器" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

    </p>
  </div> 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>